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[57] ABSTRACT 

A timing control mechanism designed for a high perfor- 
mance network wherein many of the timers are cancelled or 
reset prior to expiring. This method and apparatus enable the 
cancels and resets to occur with minimal impact to the 
system performance. An intermediate work queue is used 
where timer events are posted prior to being scheduled into 
the timer queues by the timer process. Thus, many of the 
cancels or resets will be implemented simply by changing 
the status of events in this work queue, minimizing the 
impact on the timer queues and timers. The timer process 
processes the work queue before processing any timer 
events. 
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TIMER STATE CONTROL OPTIMIZED FOR 
FREQUENT CANCEL AND RESET 
OPERATIONS 

FIELD OF THE INVENTION 5 

This invention relates to telecommunications network 
timing mechanisms in software. More specifically, the 
invention relates to a multi-class periodic timer implemen- 
tation for multi-processor systems in a telecommunications 
network where the timer queuing mechanisms are optimized 1C 
to minimize the overhead required to cancel and reset 
events. 

BACKGROUND OF THE INVENTION 

l« 

High Performance Routing (HPR) is a new transport level 
protocol used in SNA/APPN (Systems Network 
Architecture/Advanced Peer-to-Peer Networking) networks. 
One of the most significant features in HPR is end-to-end 
adaptive rate based flow control(ARB), which replaces K 
window based flow control used in prior implementations of 
the SNA architecture. HPR also provides end-to-end error 
recovery and path switching mechanisms. For additional 
information on HPR see APPN High Performance Routing 
Architecture Reference, IBM publication number SV40- 
1018 which is also accessible from the world wide web at 
/pub/aiw/appo/hpr/hprS.psbin. Each HPR connection, in 
implementing the different features, can make multiple 
concurrent periodic timer requests. Further, timer requests 
for the different features can occur at different frequencies. ^ 

The introduction of the HPR function into large hosts (390 
hosts in the preferred embodiment) introduced the require- 
ment of maintaining a large number of timers (over 1000)* 
most of which are very short duration (less than 100 
milliseconds). Additional complexities were added by the 35 
host processors having multi-processor capabilities where it 
is uncertain which processor will be handling the next 
execution of the program or the next timing request Timer 
requests can be broken down into timer classes based on the 
frequency of timer requests. At one extreme, a timer class m 
can be characterized by having a low number of outstanding 
timer requests but recurring at a relatively high frequency. 
At the other extreme, a timer class can have a very large 
number of outstanding requests but with a relatively large 
interval between requests (hence, a low frequency). Many of 4S 
the timers used in HPR have the additional characteristic 
that under normal conditions, they will be reset or cancelled 
prior to expiring. This is because the timers are set to wait 
for an acknowledgement to determine whether retransmis- 
sion is necessary and under normal usage conditions, there ^ 
is no need for retransmission since there is not excessive 
congestion nor are mere hardware failures. Some implemen- 
tations of timer management in software can significantly 
increase CPU cycle consumption thus degrading overall 
system processing capability, especially when managing 55 
disparate timer classes. For these reasons, timer design is a 
critical aspect of the HPR implementation. 
In general, implementing a timer requires the provision of 
the following basic primitives: 
Start_timer: This primitive allows the insertion of a timer 60 
request into the timer data structure so that a specified 
action can be scheduled at the appropriate time. 
Schedule_jrequest: This primitive provides the mecha- 
nism to locate the timer requests) which has expired 
(when the primitive is invoked) and schedule the sped- 65 
fied action. This primitive is usually invoked as part of 
processing an interrupt from the system dock. 



2 

Cancels request: Timers are often used to provide a 
backup mechanism to processes waiting for signals 
which are delayed or lost (For example, a timer to limit 
the amount of time spent waiting for an acknowledge- 
ment before allowing retransmission to begin). 
However, most often the signal is received before the 
timer expires which necessitates cancelling or resched- 
uling the timer request 
ReseLjrequest: This primitive is a logical concatenation 
of Cancel_jequest followed by a Schedule„request It 
is primarily used for indicating that a periodic acknowl- 
edgement has been received and incrementing the 
duration to wait for the next periodic acknowledge- 
ment 

In a well-functioning HFR environment the reset func- 
tion can have a significant impact on the performance of the 
system, Hie main costs involved are: 
Cost of searching the timer chain for the timer to be 
cancelled. 

Cost of removing the TCB (Task Control Block) from the 
TCB chain. 

Cost of inserting (or re-inserting) the timer request (TCB) 

into the timer data structure. 
Cost of synchronizing the insertion and cancellation of 

TCBs. 

An optimal timer implementation for these recurring resets 
minimizes the combined cost of: 1) finding the timer in the 
TCB data structure; 2) deleting the timer from the data 
structure, and 3) inserting the timer into the data structure. 

The simplest implementation of a timer structure, as 
shown by the prior art in FIG. 1, is to have a linked list of 
TCBs, ordered according to their time of expiration. FIG. 1 
shows a task control block (TCB). 101, which represents a 
timer which expires at the current time (T c ). The next TCB, 
102, represents a timer which expires x milliseconds from 
the current time. The TCB 103 represents the event which is 
scheduled to occur y milliseconds after the event in 102. 
This chain continues until the last scheduled event in the 
chain is reached. The sorting cost of this structure is pro- 
portional to the number of entries in the list ie. of order n. 
0(n), where n is the number of entries in the list This cost 
is incurred every time a TCB is added to the linked list since, 
as shown in FIG. 1. the list must be scanned to determine 
where the TCB is to be added. To insert an item that is 
scheduled to expire after T c+jr+y (103) and before T z (105), 
a compare must be made between the value to be inserted 
(114) and T c ; if the value is greater than T 0 it must then be 
compared with the value next in the TCB chain (T c+Jr ). This 
process is continued until the value being inserted is less 
than the value in the chain. When this occurs, the value is 
inserted before the compare value. On successive timer 
expirations, the new timer interval is set according to the 
value of the first TCB that is to expire, so that there is a timer 
dispatch with every TCB expiration. This implementation 
can lead to excessive MIPS being devoted to adding TCBs 
to the timer data structure as well as dispatching the timer 
process. Even though the sorting cost can be reduced to 
0(log(n)) by using data structures such as heaps and left- 
leaning trees . the cost of timer dispatches (scheduling TCBs) 
cannot be reduced for this linked list implementation. 

As shown in FIG. 2. the clock-tick approach tries to 
reduce the cost of TCB insertion and the excessive number 
of timer dispatches by controlling the frequency of timer 
process dispatches. The period between timer dispatches is 
the dock-tick interval which is represented as T^. TCBs are 
inserted into the TCB linked list by adding the TCBs to the 
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end of the TCB chain. There is no searching done prior to (FSM) to control the reset and cancel functions of the timer, 

insertion, a pointer is utilized which points to the end of the FIG. 3 is an example of a communications network. Two (or 

TCB chain. In FIG. 2, T f (201) was the first TCB inserted more) host computers (301302) are connected by a com- 

into the TCB chain, then T g (202) was inserted, next T* munications line (303). Each of the host computers has one 

(203) was added, then T ( and continuing onward to T„. At 5 or more consoles or terminals (304305) attached to it It is 

predetermined clock-tick intervals. T A . the linked-list of obvious to one skilled in the art that this example is an 

TCBs is searched. Each TCB which expires on or before the extremely simplistic view of a network. Typical networks 

current time is dispatched for execution. Any TCB which are much more complex. This view is for explanation 

expires after the current time interval is decremented by the purposes only and is not meant to limit the invention in any 

clock-tick interval (T A ). Hence, if the clock-tick interval 10 way. 

were 5 ms. and the TCB linked-list contained entries expir- fig. 4 shows a more detailed view of the interworkings 
ing at 10 ms. 3 ms. 1 ms. 20 ms; after the next clock-tick 0 f the HPR communications in the host computer. The 
the TCB linked-list would contain the following elements: 5 appUcation layer (401) communicates with the remote appli- 
ms. 15 ms. The cost of updating the TCBs once every cation over one or more sessions (411). The sessions are 
clock-tick interval is 0(n) since each of the TCBs must be 15 managed by the session layer (403). One or more sessions 
updated. However, the cost of inserting a new TCB is can be multiplexed over an HPR connection (413). Multiple 
reduced to 0(1) since TCBs are always added to the end of HPR connections can exist between this host and other hosts 
the TCB chain. The clock-tick interval is designed to occur OT no & cs i n the network. The HPR connection represents an 
less frequently than a TCB insertion and is typically chosen end-to-end communication capability between two end- 
to be a relatively large value to reduce the cost of timer 20 points in the network. The network layer (405) manages 
dispatches and the cost of updating the TCBs. This intro- HPR connections. Physical connectivity between the host 
duces a certain amount of imprecision in the scheduling the computer and the adjacent device is managed by the physi- 
actions associated with a TCB since the TCBs are only caj laycj (407) by using the physical link (415). The end- 
examined every clock-tick interval. The maximum error that to-end connection may involve several devices which are 
can occur is at most the clock-tick interval. 25 physically connected. 

Although the clock-tick method improves the The component of HPR responsible for controlling many 

performance, neither of the above methods solves the per- of mese ^ h mc ^ Transpor t Protocol (KIT) 

formance problems relating to a high frequency of reset com ^ ncnt h Wcamly me performance and 

commands. A new approach was needed to solve this ^ of ^ the RIP component does not have 

problem. 30 to keep track of what timers are outstanding when issuing a 

BRIEF DESCRIPTION OF THE DRAWINGS new request, thereby reducing the amount of overhead 

required. 

FIG. 1 describes the prior art linked list timer structure. 

FIG. 2 describes the prior art clock-tick method of timer 35 PREFERRED EMBODIMENT 

implementation. The HPR timers in the preferred embodiment are main- 

FIG. 3 shows an example of a communications network tained in a clock- wheel structure as described in a copending 

FIG. 4 graphically represents the internal workings of an application named **A Multi-stage Timer Implementation for 

HPR host computer. Telecommunications Transmission** by Balachandar Rajara- 

FIG. 5 shows a multiple timer set structure using timing 40 man and Subir Varma hereby incorporated by reference. In 

wheel and linked lists structures. ^ timin 8 whcd structure is a set of pointers to linked 

, . . _ . „ . lists of TCBs. Each of the pointers represents a predeter- 

FIG. 6 shows the prior art flow of a cancellation. ^ quamity of ^ ^ a ^ m ^J^ tfa e 

FIG. 7 shows the flow of a cancellation using multfcle timing wnec ^ a pointer to its TCB is placed into the linked 
TCB S * 45 Ust for me appropriate slot of tirrc^ 

FIG. 8 represents the set flow using the present invention. wheel granularity was 5 milliseconds, and the task to be 

FIG. 9 further represents the flows of a set using the scheduled was to be executed 15 milliseconds from the 
present invention. current time, then the task would be put in the linked list for 

FIG. 10 shows the states of the finite state machine (FSM) timer slot This is represented in FIG. 5. 

used in the preferred embodiment of tfae present invention. 50 HG. 5 uses three timer sets for illustration purposes. The 

FIG. 11 Depicts a visual description of the scheduling first and second timer sets (502 and 512) are organized as a 
process of the preferred embodiment. ^t Tht set 502) contains a pointer 503) 

t7t/-i „ . A f . ... r 1 j to a linked list of TCBs (501). The second timer set (512) 

FI012 shows the reUtivc simplicity of a cancel and set also a mtcr (S13) to a hDkcd ^ of TCBs (5U) 

using the present invention. 55 ^ rf ^ ^ mQ timersets(50 2 & 512) also contain the 

OBJECTS OF THE INVENTION time interval that the timer set represents (504 & 514) and 

an indication of the remaining time (505 & 515). The third 

One object of this invention is to minimize the perfor- timer set (522) in this illustration has an indication of the 

mance impact (both storage and MIPs) as the number of time interval (524) and the remaining time (525), but rather 

timer reset requests increases. Another objective is to mini- 60 than having a pointer to a linked list of TCBs. the third timer 

mize the impact of a timer cancel request on the system. sc t (522) has a pointer to a timing wheel (523). The timing 

SUMMARY OFTHE INVENTION * p^^ 00 iam f taU . ^ 

(550-555). Each of these slots represents an instance of the 

This invention discloses an efficient implementation to timer expiring. The slots (550-555) contain pointers to 

reset timer requests for a high frequency timer used in a 65 TCBs which are scheduled to expire during the increment of 

computer network The invention includes a system and time which they represent There is no significance as to how 

method for using multiple pointers and a finite state machine many or the order of the tuning wheels and the linked lists 
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(ie. the entire tuning control mechanism could use timing .... 6 

wheels or linked lists or intmnix themin ZoX 2*S ^ C °^ S -^ esi,uadoDS ^™8«h«eormore 

^entiUustn.to.ti.eTCBsrepresentedasM^^ TC f* ^ te we - * » C 

uled to exp^e durkg the time represented by slot 55 * rZ 1™"^ Md . * i yS ' em "Pacing performance 

TbelOTi^ssisusMfameprefOTrflh^ problems could encounter the need for more Ihanthree This 

fte^swrnchisusingthea^S^^^^ 5 ^Te?™^ ^cant problem when 

stalled in the art. this invention is applicable to aTprS XLwff h ° StS . may requiw over HMO timers each 

application which sets, cancels aVd/or resets 2eT£,, «anaguig multiple TCBs. 

KTP process, which requests the timerTerwTe S J^ST "J"*" deScribes * ^«hod that aUows the 

fmer block for each timer type to »Ti»™E^ ,™ ° f * e ^ f ^<* e ^P^s«s intake) to 

traditional method of doing thiis to aLK™ liS 10 r?V W - toisara ^ U ^ on Wcit We xplictt)and 

t^toratotetojj^yjj^ £ -* a « to » new duration, without thTstorage SlSS 

need to be searched and the desired TCBremoved fromT ! 1,115 a"""- »« concept of a work queue must 

q«eue. To reset the timer once it wlsZZZTZ « ^ te " ndeK ^^ Me wapioc^sortaskiss^«S 

would need to be " c^eSu^T * 25S 

operation so that it could not be used scheduled t ST* * ° D . My 0ther " ueue - ™ e ««t time that toe 

cancelled, then it would need to be ' W ZZ ,k. tmer P"**" control (is scheduled time to use 21 

^^^m^T^u^^^^ processw >- " searches its work^ Md to*Je 

processing poweTin WoZZSfflSM^ EST^r?* d ~ X£S> * 

complexities. The present invention avo^Xe " £ ^ *" WOrk qUeue has »«j««J 

mentioned problems as described below. * raEE/E?" "* toks ^odated witt 4e 

. In a multi-processor environment the number of compet S u.t L.n LT * «* toks 

^f'^^ - *tfbVI scenario of the Resent 

Furthermore, it is often necessary to start a newT™t „ Sim * ( f 2" T"™ fte "^Process (Ml) issues a set 
request (SRQ) timer (or any other tiiMrtat i 30 • * Se " ds 1,16 ,ocation of theTCB fcr that timer «o tf« 

^xir^ * s«s its te^r ss (9oi) issues • skes 

process so that it may return control of the TCB<m\i^Z Jf WUe y CB1 A pointer to TCB1 is then olacednn 

edges or replies (615) to the request of the RTP nmce^wm • 10 Mdlcate what action is to be taken. At time T a! 

by sending me TCB (MS). Once th7wpS«^ } taBer ^ 8* control. TheWk m2f«&^ 

returned TCB, it may issue the r^uesaorVrfew^a*^ Which causes *e pointer to TCTl to be o^h 

TheDroblmtathJsrctormisa^tfte^Z^ ^ lntoitea «^ rl ^*lotmthe^ e wheeH9i^ 

stars ~^*+set£* 45 s^^ssa^ 

may be delayed while waiting for the dispawTrf *S ™T '" mng whed ^ 
pr^ss).AdditionaldeIaymayl*mtrc<lucXna^^^^^ nG v W ^*eluiite state ni achine(FSM ) which act, 

.Sn^^^^^^^'-toallocate " J^n SiSf S° "* *" iS Sthe ™S 
nr«ic, ™^ ^ f""* 58 - 10 Ws scenario, the RTP ii»„if (1M1) u ,ssue4 - (he bits in the TCB are 

002 . This requires that the RTP process mainZu^ °» *e work queue and the action o be 

TCBs rather than just one or maurinTS^eSk £t wo * 1«cue is processed is mat tte fa 

actual execution. As soon as ^ 10 ^ remail,s « B»e ON QUEUE. SETH MJ\ ctlt S KM 

changed from set to cancel. Still no processing of the work 
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* e °N QUEUE- CANCEL (1003) state and ttettaer ,„.^ WOrk queue (U0S > * I*"™* by the timJpr^s 

RESET 1W1) state.If the FSMis la the ON QUEUE. SET pn i ce " whichpoints to meTOi (1110 

(1 WQ i Mb and the timer control process exeiute " UW) .„ M fl " (h \ appro P riate «*» *« *e control FSM (1115) are seTto 

&ET (1004) state means that the TCB has b«n un P rovemellt Wlth this invention man neither the nWr 
"«d into a timer slot with a specific tiineWrXe ^?™ &t ™^<^»*JmM^Z 

^ rf«Pi«tion (1020) is received. theTCB is ^^^P^^RG U^owsmerehtivesimoBcirv 
? ?«22S2!? 5 0the wwk ^««withmenewduratfo* ? f 8 «t instruction when JJTteSS 

M*«J! when the ESM b inteON B JT?? T ° te *»* * e «» Proce^lM C 

sstfsasaassassajsS ssw?sir3^- 

executes (1023). the FSM is processed from the workout SuK * work 9«eue is always processed hrfmT a- 2 - 
* ' *> the ON SLOT. SET (1004) ^^ 2* **• "* wil] be 

^"l** 0N QUEUE ON SLOT! CANCEL (££ ^ * "* ^5 St 
state and the timer control process executes (1024) A. TCB resets) * e s «"* TCB (1225) prior »7Z £ti 
^ed from the *ning whed ^ JSS?J2 40 ""ft ^ FSM state ?Zg * J 

Sir' f CIfu « FSM is in U, e ON SLOT. SET SET < 122< > without the clever betog voccSl 

(MM) state and a cancel (1025) a received, the TPB i. ^ *e timer process. This FSM would inco-l!!^, 

timer. The task is scheduled to execute and the FSM h 10 was schedule <» f <* expiration the task ^JI^/? ^ 
ma^mem a single TCB (in the terminology of the a ^Ipwess; 

requested by a specific iSta^KSS " "J** ** issuing set cancel and 
anbodul f"- this process is the OTP proem The r^ xn !* ^? mands ** one or more tasks, each of s!2d 

^(m^supr^fourtypesofame^SeSerrel wh^^J^^ **" 8 M Mo* 

embodiment they are UvenesCre-fTThort wherein each of said control blocks includes an area" 

burst) One skilled in the art wiU rejTs^S ^ £f J* S ** Storcd differed 

invention is not limited to the RTP process tn mro ■ ^ state machine; 6 res 01 a 

rh^l The . interaCt !l' n With of TCBs is ftesW « **« • P 0 ^ to me control block on the 

before only one of the TCBs (LUO) will be used TZ, ™£ m nspoast 10 ^ command issuance if 
^.rnem^proce^UO^aUtaaSt:^ ^ni^^^^^ZTtZ 



02/09/2004, EAST version: 1.4.1 



5,768,572 



means for updating said control block pointed to from 
^dworkquewkresponsetosaidcom^^cr 

"TJZt^l *»* ^ herein said 5 

S •, fim . pr0CCSsin « said Mocks 
pointed to by said pointers on said work oueue tiZ 

processing said timer structure. fl 

2. A computer network, comprising: 

two or more computers; J0 

te «~ Cati ° n£ Bnks two or more 

a timing control mechanism for optimizing timer ooera 
ta related to using said WeEZSSE 
said timing control mechanism comprising 15 
a work queue; mwug. 
one or more timing structures; 
a timer process; 
a control process; 

means for scheduling a task by placing an entry rer, * 
resentwg said task onto said work queue: ^ ^ 

means for cancelling a task by updating said entn, ™ 
«queue. when J.TZTi^Z 
entry representing said task: ""uusan 
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me^forresetogataskbyupdatingsaidentryonsaid 
work queue, when said work queue contains an enlry 
representtng said task; and 

^** f " F ^8^«« from said work queue 
prior to processing any of said timing structures 
3. A computer, comprising: 
one or more tdecommunications links; 
mews for transmitting over said links, said means for 

emitting utilizing a timer control mechS £ 

a timer structure; 
a work queue; 
a timer process; 
a control process- 

T«e7a^ to8S ^ qUeUedtoo »^*ork 

means in said timer process for proces SU i 8 said ele 
ments from said work auaur^Z, 8 . eIe * 
tuner smicti« qUepn0r,Oprooessin 8 s ^ 
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